Vehicular vision system

ABSTRACT

A vehicular vision system includes a camera, a distance sensor and a controller having at least one processor, where image data captured by the camera and sensor data captured by the distance sensor are processed at the controller. The controller, responsive to processing of captured image data and of captured sensor data, detects an object. The controller determines the distance to the detected object based at least in part on difference between the positions of the detected object in captured image data and in captured sensor data. The controller, responsive to processing of captured image data and of captured sensor data, and responsive to the determined distance to the detected object, determines that the detected object represents a collision risk. The controller informs a driver of the vehicle of the collision risk and/or controls the vehicle to mitigate the collision risk.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/349,011, filed Nov. 11, 2016, now U.S. Pat. No. 10,106,155,which is a divisional application of U.S. patent application Ser. No.13/384,673, filed Jun. 27, 2012, now U.S. Pat. No. 9,495,876, which is a371 national phase filing of PCT Application No. PCT/US2010/043363,filed Jul. 27, 2010, which claims the filing benefit of U.S. provisionalapplication Ser. No. 61/228,659, filed Jul. 27, 2009, which are herebyincorporated herein by reference in their entireties.

FIELD OF THE INVENTION

The present invention relates to a camera for use in vehicles, and moreparticularly rearview cameras for use in vehicles.

BACKGROUND OF THE INVENTION

A typical camera for mounting on a vehicle has a lens member, an imagingelement, a circuit board and housing members that connect together. Thetypical vehicular camera, however, is relatively restricted in terms ofits processing capabilities.

SUMMARY OF THE INVENTION

In one aspect of the invention, a vehicular camera is provided,comprising a lens, a housing, an imager and a microcontroller that iscapable of handling certain functions, such as applying overlays toimages received by the imager, dewarping the image and/or providingdifferent viewing modes.

In a first aspect, the invention is directed to a vehicular camera,comprising a lens, a housing, an imager and a microcontroller that iscapable of handling certain functions, such as controlling theapplication of overlays to images received by the imager, dewarping theimage and/or providing different viewing modes.

In a second aspect, the invention is directed to vehicles and/orvehicular systems that incorporate the aforementioned camera. Suchsystems include, for example, vehicle surround view systems and objectdetection/collision avoidance systems.

In a third aspect, the invention is directed to a vehicular camera,comprising a lens, an image processor and a microcontroller. Themicrocontroller is connected to the image processor by a first busthrough which command data is communicated to the image processor fromthe microcontroller. The command data includes application instructionsto draw application data from a selected point in the flash memory. Themicrocontroller is connected to the image processor by a second busthrough which application data is communicated to the image processorfrom the flash memory.

In a fourth aspect, the invention is directed to a vehicular camera thatdewarps an image by stretching and compressing portions of the image soas to provide a selected shape to a selected known element in the rawimage received by the camera. For example, the camera may be configuredto dewarp images based on the shape of the horizon line in the images.In the raw images received by the camera the horizon line may be curvedinstead of being straight as it should be if there were no warpage inthe raw images. The camera will stretch and compress portions of theimage in order to at least partially straighten out the horizon linethereby dewarping the image.

In a particular embodiment, the camera includes a lens having a field ofview, an image sensor, a housing and a controller. The lens and imagesensor are mounted to the housing at a selected position relative toeach other. The imager receives raw images from the lens. For at leastone raw image the controller is programmed to generate a dewarped imageby stretching and compressing portions of the raw image so as to providein the dewarped image a selected shape for a selected known element inthe field of view of the lens.

In a fifth aspect, the invention is directed to a camera system for avehicle, including a camera, a distance sensor, and a controller thatdetermines whether there is an element in the fields of view of thecamera and the distance sensor that represents a collision risk for thevehicle based on an image from the camera and input from the distancesensor. Upon detection of an element that represents a collision riskthe controller is programmed to carry out at least one action consistingof: informing the driver of the vehicle of the collision risk; andcontrol at least one vehicle component to inhibit a collision by thevehicle with the element that represents a collision risk.

In a sixth aspect, the invention is directed to a camera system for avehicle, including a plurality of cameras each of which is connected toa central controller by an electrical cable. Each camera has sufficientindividual processing capacity to be able to modify raw images andgenerate processed images. The processed images are sent to the centralcontroller, which combines the images into a single image which showsthe combined field of view of the plurality of cameras, and which mayoptionally apply an overlay to the single image. Because each camera hassufficient processing capacity to modify the raw images, a relativelyinexpensive type of electrical cable can be used to transmit the imagesfrom the cameras to the central controller. By contrast, in some priorart systems, the cameras are not capable of modifying the images and sotypically expensive, well-insulated cable (e.g., coaxial cable) is usedto transmit images from the camera to a central controller, whichreceives the images and carries out functions, such as dewarpingfunctions, on the images. The coaxial cables, however, are relativelyexpensive, and are difficult to route easily through the vehicle due totheir thickness and consequent low flexibility.

In a particular embodiment, the camera system includes four cameras. Thecameras are positioned at selected positions about the vehicle such thatthe combined field of view of the four cameras is substantially 360degrees. Each camera includes a lens, an image sensor, a housing and acontroller. The lens and image sensor are mounted to the housing at aselected position relative to each other. The image sensor receives rawimages from the lens. The controller is programmed to modify at leastone raw image to produce a processed image. The central controller isprogrammed to combine the processed images from the cameras into asingle combined image showing the combined field of view.

In a seventh aspect, the invention is directed to a camera for a vehiclehaving a towing device, including a lens having a field of view, animage sensor, a housing and a controller that processes raw imagesreceived by the image sensor into processed images. The camera ispositioned at an actual viewing angle and has the towing device is inthe field of view of the lens. The camera has a bird's eye viewing modein which the controller is programmed to modify the raw image so thatthe processed image appears to have been taken at an apparent viewingangle that is more vertically oriented than the actual viewing angle.

In an eighth aspect, the invention is directed to a camera for avehicle. The camera is programmed to recognize a selected feature in itsfield of view and to apply an overlay proximate the selected feature inan initial raw image showing the selected feature. As the vehicle moves,and the selected feature moves in the field of view of the camera, thecamera holds the overlay in a fixed position, so that the selectedfeature moves relative to the overlay. This can be used for severalpurposes. One purpose in particular is to assist the vehicle driver inbacking up the vehicle when towing a trailer. The selected feature wouldbe provided on the trailer (e.g., a sign with a cross-hairs on it).Initially, when the driver has the trailer directly behind the vehicle,the camera could be activated by the driver to apply an overlay to theraw image showing the cross-hairs. The overlay could be, for example, adot at the center of the cross-hairs. As the driver backs up the vehicleand trailer, the cross-hairs on the trailer will move relative to thefixed dot on the screen if the trailer begins to turn at an anglerelative to the vehicle. Thus, the driver can use the dot and thecross-hairs as a reference to keep the vehicle and the trailer straightwhile backing up.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will now be described by way of example only withreference to the attached drawings, in which:

FIG. 1 is a schematic illustration of a camera in accordance with anembodiment of the present invention;

FIG. 1A is a schematic of a camera in accordance with an embodiment ofthe present invention, illustrating electrical components associatedwith the camera;

FIG. 2 is a graph showing temperature-related specifications for thecamera shown in FIG. 1;

FIG. 3 is a table showing fatigue-related specifications for the camerashown in FIG. 1;

FIG. 4 is a view of the lens superimposed with the imager for the camerashown in FIG. 1;

FIGS. 5a, 5b and 5c are images taken using the camera shown in FIG. 1with different amounts of distortion correction;

FIG. 6 is an image from the camera shown in FIG. 1, when in across-traffic viewing mode;

FIG. 7 is an image from the camera shown in FIG. 1, when in a bird's eyeviewing mode;

FIG. 8 is an image from the camera shown in FIG. 1, showing a staticoverlay applied to the image;

FIG. 9 is an image from the camera shown in FIG. 1, showing a dynamicoverlay applied to the image;

FIG. 10 is an image from the camera shown in FIG. 1, when the camera isin a hitch mode, showing a dynamic overlay applied to the image;

FIG. 11 is a schematic illustration of a vehicle surround view systemthat uses a plurality of the cameras shown in FIG. 1;

FIG. 12 is an image taken using the vehicle surround view system shownin FIG.

FIG. 13 is an image from the camera shown in FIG. 1, during a procedureto align the lens and imager;

FIG. 14 is an image from the camera shown in FIG. 1 when the camera isin a trailer mode; and

FIG. 15 is a schematic illustration of a camera system that is used forobject detection using the camera shown in FIG. 1.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1A is a system block diagram of a camera system 100 which includesan image processor 102 operatively connected to a camera lens 103 (asdiscussed in greater detail below) and a microcontroller 104. In thepreferred embodiment the image processor 102 is a model MT9V126 whichincludes a CMOS image sensor and some control elements and is availablefrom Aptina Imaging, San Jose, Calif., and the microcontroller 104 is aPIC microcontroller available from Microchip Technology, Chandler,Ariz., and which includes an internal flash memory 110. The imageprocessor 102 and microcontroller 104 communicate via a serial, twowire, inter-integrated circuit (I²C) bus 106, as known in the art perse. The image processor 102 and microcontroller 104 also communicate viaa four wire serial peripheral interface (SPI) bus 108, as known in theart per se. The system 100 also includes a transceiver 112 that enablesthe microcontroller 104 to communicate with other devices over thevehicle internal bus 114 such as a CAN or LIN bus as known in the artper se. The image processor 102 provides a composite video output 116,such as a single ended or differential NTSC or PAL signal.

In an embodiment not shown in FIG. 1A, the image processor SPI bus isused to connect to an external flash memory device that stores overlaydata to be superimposed on the images obtained from the camera lens 103.In the embodiment shown in FIG. 1A, the SPI bus 108 is directlyconnected to the microcontroller 104, which includes I/O pins (notshown) and appropriate interface circuitry for implementing the SPI busprotocol, all under control of the microcontroller. The SPI protocolprovides for the reading or writing to a block of memory, commencingfrom a memory address. Thus, the microcontroller 104 is able to emulatea flash device, providing data from its internal flash 110, ordynamically calculating and providing imager data such as overlay datato the internal SPI bus registers for transfer to the image processor102. Thus, overlay data can be dynamically calculated on the fly. Thisprovides an advantage over the prior art where conventionally, theoverlay data had to be pre-programmed or pre-considered to account forall eventualities. For example, in an application which superimposes asquare block at any point in the image to represent a moving object, theprior art required that the flash memory be pre-programmed with a seriesof blocks that covered the entire display area, to thus mimic themovement of the block. This can be avoided by dynamic overlay datageneration by the microcontroller 104. For example, the microcontroller104 may be programmed to calculate at least one of the size and shape ofan overlay, to store the calculated shape of the overlay in its flashmemory 110 and to transmit data relating to the overlay to the imageprocessor 102 for the image processor 102 to apply to one or moreimages.

The I2C bus 106 is used by the microcontroller 104 to send command datato the image processor 102, including for example memory instructionsfor where to draw application data including for example, overlay data.The SPI bus 108 is used to communicate the application data between themicrocontroller 104 and the image processor (specifically between theflash memory 110 contained on the microcontroller 104 and the imageprocessor 102).

The implementation of the emulated flash provides a convenient methodfor the camera system 100 to access imager-specific data, includingcustom image settings, overlays, and digital correction algorithms. Theimager-specific data is organized into a series of records, includingcustom register settings sets, digital correction algorithms, overlays,and imager patches. Each record is organized into tables indexed by atable of contents. Each table is in turn indexed by a master table ofcontents. Furthermore, initialization tables are made available forcustom initialization routines.

Thus, other advantages flowing from the invention includes the fact thatflash drivers do not need to be developed to support an external device.Rather, the microcontroller has a means for performing flash-basedoperations on program memory. The bootloader also has reduced complexityas the microcontroller does not need to maintain a separate flash driverfor an external flash. There are also a reduced number of physicalconnections/communication channels. With emulated flash, a single SPIcommunication channel exists between the microcontroller and imageprocessor. With an external flash, an additional SPI connection betweenthe microcontroller and external flash would be required, in order toallow for re-flashing.

Reference is made to FIG. 1, which shows a camera 10 in accordance withan embodiment of the present invention. The camera includes a lensassembly 12, a housing 14, which may include a lens holder 14 a and arear housing member 14 b, an imager 16 and a microcontroller 18.

The lens assembly 12 is an assembly that includes a lens 22 and a lensbarrel 24. The lens 22 may be held in the lens barrel 24 in any suitableway. The lens 22 preferably includes optical distortion correctionfeatures and is tailored to some extent for use as a rearview camera fora vehicle. The distance between the top of the lens 22 and the plane ofthe imager is preferably about 25 mm.

The lens barrel 24 may have either a threaded exterior wall or athreadless exterior wall. In embodiments wherein the exterior wall isthreaded, the thread may be an M12x0.5 type thread. The barrel 24 ispreferably made from Aluminum or plastic, however other materials ofconstruction are also possible. The lens barrel 24 preferably has a glueflange thereon.

A lens retainer cap, if provided, preferably has a diameter of about 20mm. The imager 16 is preferably a ¼ inch CMOS sensor with 640×480pixels, with a pixel size of 5.6 micrometers×5.6 micrometers. The imagerpreferably has an active sensor area of 3.584 mm horizontal×2.688 mmvertical, which gives a diagonal length of 4.480 mm.

The field of view of the camera 10 may be about 123.4 degreeshorizontally x about 100.0 degrees vertically, which gives a diagonalfield of view of about 145.5 degrees. It will be understood that otherfields of view are possible, and are described further below.

The F number of the lens is preferably about 2.0 or lower.

The relative illumination is preferably >about 50% inside an imagecircle having a diameter of about 4.480 mm.

The geometrical distortion is preferably better than about −47.

The modulation transfer function (MTF) values for both tangential MTFand sagittal MTF are preferably greater than about 0.45 at 45 lp/mm onthe lens axis, and greater than or equal to about 0.30 at 45 lp/mm offthe camera axis between about 0 degrees and about 60 degrees. Exemplarycurves for the MTF value at 0 degrees and at 60 degrees are shown inFIGS. 2a and 2 b.

The lens 22 preferably has an integrated infrared cutoff filter. Thefilter can be directly coated on one of the lens elements.Alternatively, the filter can be an add-on thin glass element with thecoating thereon. The pass-band wavelengths of the infrared cutoff filtermay be about 410 nm for a low pass, and about 690 nm for a high pass.

The infrared cutoff filter preferably has at least about 85%transmission over the pass-band spectrum, and at least about 50%transmission at both 410 nm and 690 nm.

The lens 22 preferably has an anti-reflective coating on each surface ofeach lens element, except the surface with the infrared cutoff filterthereon and except any surfaces that are cemented to the lens barrel orto some other component.

The image circle diameter of the lens is preferably less than 4.80 mm.The angle between the optical axis of the lens 22 and the referencediameter axis for the barrel 24 is preferably less than 1.0 degrees.

Preferably, the lens 22 is substantially free from artificial opticaleffects, such as halo, veiling glare, lens flare and ghost images.

Preferably, the lens 22 has a hydrophobic coating on its outer surfaces,for repelling water and for providing a self-cleaning capability to thelens 22.

Preferably, the lens 22 is capable of withstanding the followingconditions without any detrimental effects, such as peel-off, cracking,crazing, voids, bubbles of lens elements, dust/water/fluid ingress,moisture condensation, foreign objects, cropping of field, non-uniformimage field and distortion that did not exist prior to the conditions,or any visible and irreversible change to the appearance of the lens 22including color and surface smooth level, or any of the aforementionedlens properties changing outside a selected range.

The conditions to be withstood by the lens 22 include:

-   -   Subjecting the lens 22 to a temperature of anywhere from −40        degrees Celsius to 95 degrees Celsius;    -   Enduring 1000 hours at 95 degrees Celsius;    -   Cycling of the lens a selected number of times between −40 and        95 degrees Celsius, with a dwell time for the lens in each        temperature of at least a selected period of time and a ramping        time to reach one or the other temperature of less than another        selected period of time;    -   Exposing the lens 22 to 85 degrees Celsius and 85% relative        humidity for 1200 hours;    -   Subjecting the lens 22 to 10 cycles of the test profile shown in        FIG. 2, including a first set of 5 cycles that include exposing        the lens 22 to the portion of the cycle wherein the temperature        drops to −10 degrees, and then a second set of 5 cycles wherein        that portion of the cycle is replaced by ramping the lens 22        from 45 degrees Celsius to 25 degrees Celsius;    -   Dropping a lens with a front cover thereon from a selected        height, such as 1 m, onto a concrete or steel surface;    -   Running the lens 22 under 6 shock pulses with 100 g and 10 ms        half-sine pulses, one in each opposite direction of 3        perpendicular axes;    -   Exposing the lens 22 to a vibration test for 27 hours in the        X-axis, 27 hours in the Y-axis and 81 hours in the Z-axis, with        a RMS acceleration value of 27.8 m/s² according to the power        spectrum density/frequency table shown in FIG. 3;    -   Exposing the lens 22 to the following test procedure relating to        abrasion resistance: Prepare 1000 ml of sludge in which 100 g of        Kanto loam (JIS Z 8901 class 8) is dissolved. Soak a sponge in        the sludge and scrub the lens surface with the sponge for 250        repetitions, applying 20N of pressing force while scrubbing.    -   Exposing the lens 22 to heavy splash shower, car wash spray mist        and dust, with the condition that any moisture condensation in        the lens assembly 12 dissipates within 10 minutes;    -   Exposure to water without leakage therepast in accordance with        ISO standard 20653;    -   Meeting IP Code 7 and 6K requirements;    -   Heating the lens 22 to 85 degrees Celsius for 1 hour, spraying        the upper body of the lens 22 with 1200 psi water stream for 2        minutes at a 0 degree angle, a 90 degree angle and at a 45        degree angle relative to the lens axis, respectively;    -   Soaking the lens in water at 95 degrees Celsius for 1 hour and        then immediately dunk the upper body of the lens 22 in room        temperature water at a depth of 15 cm for 30 minutes;    -   Meeting ISO Standard 20653 with respect to dust requirement, and        meeting IP Code 6K for protection against foreign objects. For        the testing a vertical flow chamber is preferably used, with a        dust density of 2 kg/m³ for 8 hours continuous circulation.    -   Exposing the upper part of the lens to 5% salt water at 35        degrees Celsius for 96 hours;    -   Testing that the front glass surface of the lens 22 and the        upper exterior body of the lens 22 resist the following        chemicals: Automatic transmission fluid, hypoid lubricant,        hydraulic fluid, power steering fluid, differential lubricant,        central hydraulic fluid, engine oil, engine wax protective,        engine coolant/Ethylene Glycol, gasoline, diesel fuel, kerosene,        bio-diesel/Methanol based fuel, brake fluid, windshield washer        fluid, window glass cleaner, car wash cleaner/soap solution, car        wax and silicone protectants, leather wax, battery acid—dilute        sulfuric acid (density: 1.285 g/cm³), and CaCl₂);

For each chemical tested, the test is preferably conducted for 24 hoursof exposure. The test is conducted in accordance with the followingprocedure:

-   -   1. Place test sample in a temperature chamber maintained at        40° C. on a test fixture representative of in-vehicle position        with any protective surrounding structures.    -   2. Keep test sample at 40° C. for one hour. Remove sample from        the chamber and apply 100 ml of test chemical/fluid by either        spraying or pouring to front glass surface and upper exterior        body. Store the sample at outside ambient temperature (RT) for 1        hour.    -   3. Replace the sample in the chamber and keep it at 40° C. for        one hour. Then ramp up the chamber temperature to 70° C. (60° C.        for battery acid) within 30 minutes and keep the sample at that        temperature for 4 hours (dwell time). Ramp down the chamber        temperature to 40° C. within 30 minutes.    -   4. Repeat step B and C for the same fluid but prolong dwell time        from 4 hours to 12 hours at high temperature.    -   5. Repeat step B, C and D for the next fluid in the set.        Continue the process up to a maximum of four fluids per sample.    -   Exposing the lens to a test procedure laid out in IEC 60068-2-60        method 4, whereby the lens 22 is exposed to H2S in a gas        concentration of 10 ppb, SO₂ in a gas concentration of 200 ppb,        Chlorine in a gas concentration of 10 ppb and NO₂ in a gas        concentration of 200 ppb;    -   Subjecting the lens 22 to UV exposure test as referenced to SAE        J1960v002, with a minimum exposure level is 2500 KJ/m². There        must be no significant change of color and gloss level or other        visible detrimental surface deterioration in any part of the        lens upper body. The lens upper body includes and is not limited        to lens cap surface, the subsurface under the first glass        element, top glass and its surface coating. There is preferably        no crack, crazing, bubbles or any defect or particles appear        after UV exposure in and on any of the glass or plastic lens        elements and its coatings.

The lens cap is preferably black and free of ding, dig, crack, bur,scratch, or other visible defects. There must be no visible colorvariation on a lens cap and among lenses.

The coating of the first glass is preferably free of dig, scratch,peeling, crack, bubble or flaking. The color appearance of the ARcoating should have no or minimum visible color variation among lenses.

The interior of the lens 22 (at the mating surfaces of the constituentlens elements) preferably has little or no moisture inside it, which cancause water spots on the inside surfaces of the lens 22 after cycles ofcondensation and subsequent evaporation, and which can leak out from thelens 22 into portions of the camera 10 containing electrical componentsthereby causing problems with those electrical components. Preferably,the lens 22 is manufactured in a controlled, low-humidity environment.Other optional steps that can be taken during lens manufacture include:drying the interior surfaces of the lens 22, and vacuum packing the lensfor transportation.

The manufacture of the lens 22 using a plurality of lens elements thatare joined together can optically correct for distortion that canotherwise occur. Such optical distortion correction is advantageousparticularly for a lens with a field of view that approaches 180 degreeshorizontally, but is also advantageous for a lens 22 with a lesser fieldof view, such as a 135 degree field of view. An example of the effectsof optical distortion correction is shown in FIG. 5b . The multiple lenselements that are provided in the lens each have selected opticalproperties, (e.g., refractive index), and are shaped in a selected wayto optically dewarp the image produced by the lens, as compared to asingle element, spherical lens.

Aside from optical distortion correction, the camera 10 preferably alsoprovides other forms of distortion correction. To carry this out,selected techniques may be employed. For example, one technique is toposition the lens 22 so that the horizon line (shown at 28) in the fieldof view (shown at 30) lies near the optical axis of the lens 22 (shownat 32). As a result, there will be less distortion in the horizon linein the image sent from the camera 10 to the in-vehicle display. Asidefrom positioning the lens 22 so that the horizon line is closer to theoptical axis of the lens 22, the microcontroller 18 preferably processesthe image to straighten the horizon line digitally (i.e., by compressingand/or stretching selected vertically extending portions of the image).In some embodiments, the amount of distortion in the image will increaseproportionally with horizontal distance from the optical axis. Thus, theamount of compression or stretching of vertically extending strips ofthe image will vary depending on the horizontal position of the strip.The portions of the image to stretch or compress and the amounts bywhich to compress them can be determined empirically by testing anexample of the camera so as to determine the amount of distortionpresent in vertically extending portions (i.e., vertical strips) of theimage that contain image elements that have a known shape. For example,the horizon line should appear as a straight horizontal line in cameraimages. Thus when testing the camera, values can be manually calculatedfor use to compress or stretch vertically extending portions (e.g.,strips) above and below the horizon line so that the horizon lineappears straight in the image. These values can then be used inproduction versions of the camera that will have the same lens and thesame orientation relative to the horizon line. As an alternative insteadof manually calculating the compression and stretch values to apply tovertical strips of the image, the microcontroller 18 may be programmedto carry out a horizon line detection routine, taking into account thatthe horizon line is vertically close to the optical axis in the centerregion of the image to assist the routine in finding the horizon line.It will be understood that other selected known elements could bepositioned proximate the optical axis and used instead of the horizonline.

The microcontroller 18 (FIG. 1) is preferably further programmed tostretch or compress other selected portions of the image so that otherselected, known image elements in the image appear closer to how theywould appear if the image was not distorted. For example, themicrocontroller 18 may carry out an edge detection routine to determinewhere the edge of the bumper appears in the image. Alternatively,testing can be carried out on the camera to manually determine where thebumper is in the image. Once the edge of the bumper is located eithermanually or by use of an edge detection routine, values can bedetermined to use to compress or stretch vertically extending portions(i.e., strips) above and below the edge of the bumper to adjust theposition of the edge of the bumper so that it more closely matches aselected shape (i.e., so that it more closely matches the shape itshould have if the image were not distorted). It will be understood thatthe ‘selected shape’ of the bumper edge may be a horizontal line if thebumper has a rear edge that is straight. However, the selected shape maybe a curve since many vehicle bumpers have rear edges that are curved.

Aside from reducing the amount of warping in the portions of the imagecontaining the bumper and the horizon line, the microcontroller 18 canalso modify the image in a way to make vertical elements of the imageappear approximately vertical. These digital distortion correction stepscan take place in a selected order. For example, the first step can beto straighten the vehicle bumper. A second step can be to straighten thehorizon line. A third step can be to straighten vertical objects.

In embodiments or applications wherein other artifacts are always in theimage received by the camera 10, these artifacts may be used todetermine the image modification that can be carried out by themicrocontroller 18 to at least straighten out portions of the image thatshow the artifacts.

As shown in FIG. 5c , the combination of digital distortion correctionand optical distortion correction reduces the amount of overalldistortion in the image sent by the camera to an in-vehicle display.

In addition to the distortion correction, the microcontroller 18 ispreferably capable of providing a plurality of different image types.For example, the microcontroller 18 can provide a standard viewing modewhich gives an approximately 135 degree field of view horizontally, a‘cross-traffic’ viewing mode which gives an approximately 180 degreefield of view horizontally, and a bird's eye viewing mode, which gives aview that appears to be from a camera that is spaced from the vehicleand is aimed directly downwards. The standard viewing mode (with opticaland digital distortion correction) is shown in FIG. 5 c.

The cross-traffic viewing mode is shown in FIG. 6. The cross-trafficviewing mode provides a view of the regions behind to the left andbehind to the right of the vehicle, so that a driver can determine, forexample, whether it is safe to back out of a parking spot in a parkinglot. The cross-traffic view provides the driver with an approximately180 degree view. This permits the driver to see to the left of thevehicle, to the right of the vehicle and directly behind the vehicle allat once. The digital dewarping (i.e., the digital distortion correction)that is carried out by the microcontroller 18 (FIG. 1) for thecross-traffic view is different than the digital dewarping that iscarried out for the standard view.

It will be understood that, in order to provide the cross-trafficviewing mode, the lens 22 (FIG. 1) is constructed to have a field ofview of approximately 180 degrees horizontally (i.e., it is a 180 degreelens). That same lens 22 is also used when the camera provides thestandard viewing mode with the 135 degree field of view. Accordingly,the standard viewing mode involves cropping portions of the image takenby the 180 degree lens 22.

While a 180 degree lens 22 is preferable for the camera 10, it isalternatively possible for the lens 22 to be a 135 degree lens. In sucha case, the camera 10 would not provide a cross-traffic viewing mode.

The bird's eye viewing mode (FIG. 7) can be provided by compressingselected portions of the image and expanding other portions of it,thereby moving the apparent viewpoint of the camera 10.

The bird's eye viewing mode can be used, for example, to assist thedriver when backing up the vehicle to connect it to a trailer hitch. Thebird's eye viewing mode provides a view that appears to come from aviewpoint that is approximately directly above the tow ball on thevehicle. This viewing mode is discussed further below.

In addition to providing a plurality of viewing modes, the camera 10 ispreferably capable of providing graphical overlays on the images priorto the images being sent from the camera 10 to an in-vehicle display.Preferably, the camera 10 can provide both static overlays and dynamicoverlays. Static overlays are overlays that remain constant in shape,size and position on the image. An example of a static overlay is shownat 36 in FIG. 8. The static overlay 36 shown in FIG. 8 providesinformation regarding the width of the vehicle and rough distanceinformation behind the vehicle.

Many different types of dynamic overlay can be provided for manydifferent functions. A first example of a dynamic overlay is shown at 38in FIG. 9. The dynamic overlay 38 shows the vehicle's projected pathbased on the current steering wheel angle of the vehicle. It will benoted that the overlay 38 may have a selected degree of transparency, sothat the image behind it shows through and is thus not completelyobscured.

Another example of a dynamic overlay is shown at 40 in FIG. 10. Theoverlay 40 shows the projected path of the tow ball of the vehicle basedon the vehicle's current steering angle. This overlay 40 gives thedriver useful information when backing the vehicle up to align the towball shown at 42 with a hitch (not shown). It will be noted that thecamera 10 is operating in a hitch viewing mode to provide the imageshown in FIG. 10. The hitch viewing mode incorporates the bird's eyeviewing mode, along with a digital zoom function that provides thedriver with a magnified view of the tow ball 42 and its immediatesurroundings. This view, combined with the overlay 40 gives the driverdetailed information so that the driver can align the tow ball 42 with ahitch relatively precisely. It will be understood that the tow ballcould alternatively be any other type of towing device.

Another application of the camera 10 that combines the features ofoverlays and the bird's eye viewing mode is in a 360 degree view system,an example of which is shown at 50 in FIG. 11. The 360 degree viewsystem 50 includes four cameras 10, including a front camera 10 a, arear camera 10 b, a driver's side camera 10 c and a passenger sidecamera 10 d, and a central controller 52. Each of the cameras 10 a, 10b, 10 c and 10 d can be operated in a bird's eye viewing mode and canprovide images to the central controller 52. The central controller 52can then put together an image shown in FIG. 12, with a static overlay54 which would be a representation of a top view of the vehicle alongwith the four bird's eye views shown at 56 and individually 56 a, 56 b,56 c and 56 d from the four cameras 10 (FIG. 11). The bird's eye views56 a, 56 b, 56 c and 56 d are preferably merged together into a singleimage 58 without any seam lines where one view 56 mates with another.

Because the cameras 10 are each capable of dewarping the images theyreceive, and of processing the images to provide a bird's eye view, andof adding graphic overlays on the images, the cameras 10 can be used forall of these functions and the processed images produced by the cameras10 and be sent to the central controller 52 using inexpensive electricalcables, such as shielded twisted (or untwisted) pair cables, shownschematically in FIG. 11 at 60. These electrical cables 60, in additionto being relatively inexpensive are also relatively flexible, makingthem easy to route through the vehicle between the cameras 10 and thecentral controller 52.

By contrast, if such as a system were provided with cameras that werenot themselves equipped with sufficiently powerful on-boardmicrocontrollers 18 to carry out the aforementioned functions, thefunctions would have to be carried out externally, e.g., by the centralcontroller 52. In such a situation, the raw images received by thecameras 10 would have to be sent to the central controller 52 forprocessing. A relatively great amount of care would need to be taken toensure that the raw images were transmitted to the central controller 52in relatively pristine condition, since the processing of the imageswill result in some degradation of the images. In order to minimize thedegradation of the images in their transmission from the cameras to thecentral controller, the electrical cables and any connectors could berelatively expensive (e.g., coaxial cable). In addition, the electricalcables could be relatively inflexible and difficult to route through thevehicle.

Reference is made to FIG. 1. Because the camera 10 has the capability ofapplying graphic overlays to the image using the on-boardmicrocontroller 18, the overlays can be aligned to the optical axis ofthe lens 22 before the camera 10 leaves the assembly facility. Thecamera 10 may be held in a fixture that is itself positioned at a knownposition relative to a selected reference pattern, shown at 62 in FIG.13. When in the fixture, the image received by the camera's imager 16 issent to an external controller (not shown). The external controllercompares the image received from the camera 10 with the image that wouldhave been received if the center of the imager 16 (FIG. 2) wereprecisely aligned with the optical axis of the lens 22 (FIG. 1). Theoffset between the actual and theoretically perfect images indicates theamount of offset present between the optical axis of the lens 22 and thecenter of the imager 16. This offset provides the camera 10 with anoffset value to apply to certain types of overlays when they areprovided on the image, such as the static overlay 36 shown in FIG. 8,the dynamic overlay 38 shown in FIG. 9, and the dynamic overlay 40 shownin FIG. 10. By aligning the overlay to the optical axis of the lens 22,a source of error in the positioning of the overlays is eliminated.

Reference is made to FIG. 14, which shows a trailer 64 with a targetshown at 66. The microcontroller 18 is capable of identifying selectedtarget shapes in images received on the imager 16. In the embodimentshown in FIG. 14, the target is a cross. With the trailer 64 positioneddirectly behind the vehicle (i.e., at a zero angle relative to thevehicle), the driver 10 can instruct the camera 10 to enter a ‘trailer’mode, wherein the microcontroller 18 finds the center of the target 66.Once found, the microcontroller 18 applies an overlay shown at 68 to theimage of the target 66. The overlay 68 may be a spot that appears at thecenter of the target 66. As the driver backs up the vehicle, the target66 will shift in the image transmitted from the camera 10 to thein-vehicle display, but the overlay 68 is held in a fixed position, andthus the driver will see an offset between the overlay 68 and the centerof the target 66 depending on the angle between the trailer 64 and thevehicle. By steering the vehicle so that the overlay 68 remains at thecenter of the target 66, the driver can back the vehicle up keeping thetrailer 64 and the vehicle aligned. It will be understood that theoverlay 68 need not be a dot and need not be at the center of the target66. The system will be useful even if the overlay 68 is near the target66 but not covering the target 66. Thus, the overlay 68 can be used ifit is proximate the target 66 (i.e., sufficiently close to the target 66or covering some or all of the target 66 that relative movement betweenthe target 66 and the overlay 68 is noticeable by the driver of thevehicle).

Reference is made to FIG. 15, which shows a schematic illustration of acamera system 200 that provides obstacle detection to the driver of thevehicle. The camera system 200 includes the camera 10, a cameraprocessor 202, a distance sensor system 204, a distance sensor processor206, a fusion controller 208, an overlay generator 210 and aHuman/Machine Interface (HMI) 212. The camera 10 as described receives afirst image on the imager 14 (FIG. 1). The first image has a firstresolution that depends on the resolution of the imager 14. For atypical application, the resolution is preferably 640×480 pixels. Thefirst image itself does not contain any explicit distance informationregarding the distances of objects behind the vehicle, however, thecamera processor 202 is configured to process the first image to detectobjects of interest in the first image (e.g., people, animals, othervehicles) and to determine whether such objects of interest represent acollision risk.

The distance sensor system 204 preferably includes an infraredtime-of-flight sensor 214 (which may be referred to as a TOF sensor) anda plurality of light sources 216. The light sources 216 emit modulatedlight, which reflects off any objects behind the vehicle. The reflectedlight is received by an imager 218 that is part of the TOF sensor 214.The image that is formed on the imager 218 is a greyscale image, whichmay be referred to as a second image. The second image has a secondimage resolution that depends on the resolution of the imager 218. In atypical application, the second image resolution will be lower than theresolution of the first image with the camera 10. The first and secondimages may be processed by the fusion controller 208 to generate astereo image, which provides the fusion controller 208 with depthinformation relating to objects in the two images. The fusion controller208 uses the depth information to determine if any objects behind thevehicle represent a collision risk, in which case the fusion controller208 determines what action, if any, to take. One action that can betaken is for the fusion controller 208 to send a signal to a vehiclecontrol system (shown at 219) to apply the parking brake, or the regularvehicle brakes or to prevent their release. Additionally, the fusioncontroller 208 communicates with the overlay generator 210 to apply anoverlay on the image warning the driver of the object or objects thatrepresent a collision risk. The overlay could, for example be a boxaround any such objects in the image. The overlay generator 210 maycommunicate the overlay information back to the camera 10, which thensends the image with the overlay to the in-vehicle display, shown at220, which makes up part of the HMI 212. The rest of the HMI may be madeup of a touch screen input 221 that is superimposed on the display 220.In order to override the system 200 and release the applied brake, thedriver can interact with the HMI 212 to press a selected on-screenbutton to indicate to the system 200 that the objects have been seen andthe driver does not feel that they represent a collision risk, at whichtime the system 200 can release the applied brake. In addition to theoverlay, the driver of the vehicle can be notified of a collision riskby way of sound (e.g., a chime, a beep or a voice message) or by way oftactile feedback, such as by vibration of the steering wheel or seat.

Additionally, the image received by the imager 218 can be processed bythe distance sensor processor 106 to determine the phase shift of thelight at each pixel on the imager 218. The phase shift of the light isused to determine the distance of the surface that reflected the lightto the TOF sensor 214. Thus, the distance sensor processor 206 cangenerate a depth map relating to the image. Optionally, the distancesensor processor 206 processes the pixels in groups and notindividually. For example, the processor 206 may obtain average phaseshift data from groups of 2×2 pixels. Thus, the depth map has a thirdresolution that is lower than the second image resolution.

The depth map is sent to the fusion controller 208, which can interpretit to determine if any objects shown therein represent a collision risk.The fusion controller 208 preferably works with both the depth map andthe camera image to improve the determination of whether detectedobjects are collision risks. For example, the fusion controller 208 maydetermine from the depth map, that there is an object that is within 2meters from the vehicle towards the lower central portion of the depthmap. However, the fusion controller 208 may determine from the cameraprocessor 202 that the object in that region is a speed bump, in whichcase the fusion controller 208 may determine that this does notrepresent a collision risk and so the system 200 would not warn thedriver, and would not apply the brake.

In different situations, the fusion controller 208 gives greater weightto the information from either the depth map or the camera image. Forexample, for objects that are farther than 3 meters away, the fusioncontroller 208 gives greater weight to information from the camera 10and the camera processor 202. For objects that are closer than 3 meters(or some other selected distance) away, the fusion controller 208 givesgreater weight to information from the distance sensor 204 and thedistance sensor processor 206.

The camera processor 202 is configured to recognize certain types ofobject in the images it receives from the camera 10, such as an adult, achild sitting, a toddler, a vehicle, a speed bump, a child on a bicycle,tall grass, fog (e.g., fog from a sewer grating or a manhole, or fogfrom the exhaust of the vehicle itself). In some situations, the fusioncontroller 208 determines whether there is movement towards the vehicleby any objected in the images it receives. In some situations, when thefusion controller 208 determines from the depth map that an object istoo close to the vehicle, it uses information from the camera processor202 to determine what the object is. This assists the fusion controller208 in determine whether the object is something to warn the driverabout (e.g., a child, or a tree), or if it is something to be ignored(e.g., exhaust smoke from the vehicle itself, or a speed bump).Additionally, this information can also be used by the overlay generator210 to determine the size and shape of the overlay to apply to theimage. It will be understood that some of the elements recognized by thecamera processor 202 belong to a category containing objects ofinterest, such as the adult, the child sitting, the toddler, the vehicleand the child on the bicycle. Other elements recognized by the cameraprocessor 202 may belong to a category of elements that do not representa collision risk, such as, a speed bump, tall grass and fog.

After the TOF sensor 214 and the camera 10 are installed on the vehicle,a calibration procedure is preferably carried out. The calibrationprocedure includes displaying an image with high contrast elements onit, at a selected distance from the vehicle, at a selected positionvertically and laterally relative to the vehicle. The image can be forexample, a white rectangle immediately horizontally adjacent a blackrectangle. The fusion controller 208 determines the relative positionsof the mating line between the two rectangles on the two imagers 16 and218. During use of the camera system 200 this information is used by thefusion controller 208 to determine the distances of other objects viewedby the camera 10 and the TOF sensor 214. The calibration procedure couldalternatively be carried out using a checkerboard pattern of four ormore rectangles so that there are vertical and horizontal mating linesbetween rectangles.

Throughout this disclosure the term imager and image processor have beenused. These terms both indicate a device that includes an image sensorand some control elements. The microcontroller and the portion of theimager including the control elements together make up a ‘controller’for the camera. It will be understood that in some embodiments, and forsome purposes the control of the camera need not be split into thecontrol elements in the imager and the microcontroller that is externalto the imager. It will be understood that the term ‘controller’ isintended to include any device or group of devices that control thecamera.

While the above description constitutes a plurality of embodiments ofthe present invention, it will be appreciated that the present inventionis susceptible to further modification and change without departing fromthe fair meaning of the accompanying claims.

1. A vehicular vision system, said vehicular vision system comprising: a camera comprising a lens and an image sensor, wherein the camera is disposed at a vehicle equipped with said vehicular vision system so as to have a field of view exterior of the equipped vehicle; a distance sensor disposed at the equipped vehicle so as to have a field of sensing exterior of the equipped vehicle; wherein the distance sensor comprises a plurality of infrared light-emitting light sources, and wherein the distance sensor senses infrared light; a controller comprising at least one processor, wherein image data captured by the camera and sensor data captured by the distance sensor are processed at the controller; wherein the controller, responsive to processing at the controller of image data captured by the camera and of sensor data captured by the distance sensor, detects an object present in the field of view of the camera and in the field of sensing of the distance sensor; wherein the controller determines the distance to the detected object based at least in part on difference between the position of the detected object in image data captured by the camera and the position of the detected object in sensor data captured by the distance sensor; wherein the controller, responsive to processing at the controller of image data captured by the camera and of sensor data captured by the distance sensor, and responsive to the determined distance to the detected object, determines that the detected object represents a collision risk; and wherein, responsive to determination that the detected object represents a collision risk, the controller carries out at least one action selected from the group consisting of (i) informing a driver of the equipped vehicle of the collision risk and (ii) controlling the equipped vehicle to mitigate the collision risk.
 2. The vehicular vision system as claimed in claim 1, wherein the controller receives a depth map from the distance sensor and determines distance to the detected object based at least in part on the depth map.
 3. The vehicular vision system as claimed in claim 1, wherein, responsive to determination that the detected object represents the collision risk, the controller alerts a driver of the equipped vehicle of the collision risk.
 4. The vehicular vision system as claimed in claim 1, wherein, responsive to determination that the detected object represents the collision risk, the controller controls at least one vehicle component to mitigate collision by the equipped vehicle with the detected object.
 5. The vehicular vision system as claimed in claim 4, wherein, responsive to determination that the detected object represents the collision risk, the controller controls braking of the equipped vehicle.
 6. The vehicular vision system as claimed in claim 1, wherein image data captured by the camera has a first resolution and sensor data captured by the distance sensor has a second resolution that is lower than the first resolution.
 7. The vehicular vision system as claimed in claim 1, wherein, prior to determining whether the detected object represents a collision risk, the controller determines whether the detected object falls within a category of objects that do not represent a collision risk based at least in part on processing of image data captured by the camera.
 8. The vehicular vision system as claimed in claim 1, wherein, prior to determining whether the detected object represents a collision risk, the controller determines whether the detected object falls within a category of objects that represent objects of interest based at least in part on processing of image data captured by the camera.
 9. The vehicular vision system as claimed in claim 8, wherein the category of objects that represent objects of interest include at least one selected from the group consisting of people, animals and other vehicles.
 10. The vehicular vision system as claimed in claim 1, comprising a display disposed in an interior of the equipped vehicle and viewable by a driver of the equipped vehicle, and wherein the display displays video images derived from image data captured by the camera, and wherein the displayed video images include images of the detected object that represents the collision risk, and wherein the display displays an overlay that highlights the displayed detected object.
 11. The vehicular vision system as claimed in claim 10, wherein the controller processes image data captured by the camera and processes sensor data captured by the sensor to determine a projected trajectory for the equipped vehicle, and wherein the controller applies a projected path overlay at the displayed video images, and wherein the projected path overlay comprises a representation of the projected trajectory.
 12. The vehicular vision system as claimed in claim 10, wherein the controller modifies raw image data captured by the camera to produce a processed image, and wherein the controller determines the position of a selected feature in the field of view of the camera and applies an overlay on the selected feature in an initial raw image data to produce an initial processed image, and wherein the controller thereafter holds the overlay in a fixed position in subsequent processed images regardless of movement of the selected feature in subsequent raw image data.
 13. The vehicular vision system as claimed in claim 12, wherein the selected feature is a feature on a trailer being towed by the equipped vehicle.
 14. The vehicular vision system as claimed in claim 10, wherein the camera is positioned at an actual viewing angle, and wherein said vehicular vision system has a bird's eye viewing mode in which the controller modifies image data captured by the camera so that the displayed video images appear to be at an apparent viewing angle that is more vertically oriented than the actual viewing angle of the camera.
 15. The vehicular vision system as claimed in claim 14, wherein the controller compresses a lower portion of captured image data and stretches an upper portion of captured image data so that the apparent viewing angle is more vertically oriented than the actual viewing angle.
 16. The vehicular vision system as claimed in claim 14, wherein the equipped vehicle has a towing hitch, and wherein the towing hitch is in the field of view of the camera, and wherein said vehicular vision system has a hitch viewing mode in which the towing hitch in the displayed video images is magnified to a magnification level that is greater than a magnification level provided in the bird's eye viewing mode.
 17. A vehicular vision system, said vehicular vision system comprising: a camera comprising a lens and an image sensor, wherein the camera is disposed at a vehicle equipped with said vehicular vision system so as to have a field of view exterior of the equipped vehicle; a distance sensor disposed at the equipped vehicle so as to have a field of sensing exterior of the equipped vehicle; wherein the distance sensor comprises a plurality of infrared light-emitting light sources, and wherein the distance sensor senses infrared light; a controller comprising at least one processor; wherein image data captured by the camera and sensor data captured by the distance sensor are processed at the controller; wherein the controller, responsive to processing at the controller of image data captured by the camera and of sensor data captured by the distance sensor, detects an object present in the field of view of the camera and in the field of sensing of the distance sensor; wherein the controller receives a depth map from the distance sensor and determines distance to the detected object based at least in part on the depth map; wherein the controller, responsive to processing at the controller of image data captured by the camera and of sensor data captured by the distance sensor, and responsive to the determined distance to the detected object, determines that the detected object represents a collision risk; and wherein, responsive to determination that the detected object represents a collision risk, the controller controls the equipped vehicle to mitigate the collision risk.
 18. The vehicular vision system as claimed in claim 17, comprising a display disposed in an interior of the equipped vehicle and viewable by a driver of the equipped vehicle, and wherein the display displays video images derived from image data captured by the camera, and wherein the displayed video images include images of the detected object.
 19. The vehicular vision system as claimed in claim 18, wherein the display displays an overlay that highlights the displayed detected object.
 20. The vehicular vision system as claimed in claim 18, wherein the camera is positioned at an actual viewing angle, and wherein said vehicular vision system has a bird's eye viewing mode in which the displayed video images appear to be at an apparent viewing angle that is more vertically oriented than the actual viewing angle of the camera.
 21. The vehicular vision system as claimed in claim 20, wherein the controller compresses a lower portion of captured image data and stretches an upper portion of captured image data so that the apparent viewing angle is more vertically oriented than the actual viewing angle.
 22. The vehicular vision system as claimed in claim 20, wherein the equipped vehicle has a towing hitch, and wherein the towing hitch is in the field of view of the camera, and wherein said vehicular vision system has a hitch viewing mode in which the towing hitch in the displayed video images is magnified to a magnification level that is greater than a magnification level provided in the bird's eye viewing mode.
 23. A vehicular vision system, said vehicular vision system comprising: a camera comprising a lens and an image sensor, wherein the camera is disposed at a vehicle equipped with said vehicular vision system so as to have a field of view exterior of the equipped vehicle; a distance sensor disposed at the equipped vehicle so as to have a field of sensing exterior of the equipped vehicle; wherein image data captured by the camera has a first resolution and sensor data captured by the distance sensor has a second resolution that is lower than the first resolution; wherein the distance sensor comprises a plurality of infrared light-emitting light sources, and wherein the distance sensor senses infrared light; a controller comprising at least one processor, wherein image data captured by the camera and sensor data captured by the distance sensor are processed at the controller; a display disposed in an interior of the equipped vehicle and viewable by a driver of the equipped vehicle, and wherein the display displays video images derived from image data captured by the camera; wherein the controller, responsive to processing at the controller of image data captured by the camera and of sensor data captured by the distance sensor, detects an object present in the field of view of the camera and in the field of sensing of the distance sensor; wherein the controller determines the distance to the detected object based at least in part on difference between the position of the detected object in image data captured by the camera and the position of the detected object in sensor data captured by the distance sensor; wherein the displayed video images include images of the detected object; wherein the controller, responsive to processing at the controller of image data captured by the camera and of sensor data captured by the distance sensor, and responsive to the determined distance to the detected object, determines that the detected object represents a collision risk; and wherein, responsive to determination that the detected object represents a collision risk, the controller carries out at least one action selected from the group consisting of (i) informing the driver of the equipped vehicle of the collision risk and (ii) controlling the equipped vehicle to mitigate the collision risk.
 24. The vehicular vision system as claimed in claim 23, wherein the display displays an overlay that highlights the displayed detected object.
 25. The vehicular vision system as claimed in claim 23, wherein the camera is positioned at an actual viewing angle, and wherein said vehicular vision system has a bird's eye viewing mode in which the displayed video images appear to be at an apparent viewing angle that is more vertically oriented than the actual viewing angle of the camera.
 26. The vehicular vision system as claimed in claim 25, wherein the equipped vehicle has a towing hitch, and wherein the towing hitch is in the field of view of the camera, and wherein said vehicular vision system has a hitch viewing mode in which the towing hitch in the displayed video images is magnified to a magnification level that is greater than a magnification level provided in the bird's eye viewing mode. 